<html>
<head><meta charset="utf-8"><title>tree-agnostic parsing · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic.20parsing.html">tree-agnostic parsing</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="159057283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic%20parsing/near/159057283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic.20parsing.html#159057283">(Feb 21 2019 at 10:47)</a>:</h4>
<p>I've went ahead and extracted rust-analyzer's parser to a separate crate, which depends neither on syntax trees nor on tokens: <a href="https://github.com/rust-analyzer/rust-analyzer/pull/867" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/pull/867">https://github.com/rust-analyzer/rust-analyzer/pull/867</a></p>



<a name="159057317"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic%20parsing/near/159057317" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic.20parsing.html#159057317">(Feb 21 2019 at 10:48)</a>:</h4>
<p>The interface is these two traits:</p>
<div class="codehilite"><pre><span></span>pub trait TokenSource {
    fn token_kind(&amp;self, pos: usize) -&gt; SyntaxKind;
    fn is_token_joint_to_next(&amp;self, pos: usize) -&gt; bool;
    fn is_keyword(&amp;self, pos: usize, kw: &amp;str) -&gt; bool;
}
pub trait TreeSink {
    fn leaf(&amp;mut self, kind: SyntaxKind, n_tokens: u8);
    fn start_branch(&amp;mut self, kind: SyntaxKind, root: bool);
    fn finish_branch(&amp;mut self, root: bool);
    fn error(&amp;mut self, error: ParseError);
}
</pre></div>



<a name="159057531"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic%20parsing/near/159057531" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic.20parsing.html#159057531">(Feb 21 2019 at 10:50)</a>:</h4>
<p>That is kinda cool,  because parser does not know about the underlying text, comments and whitespace (it doesn't know even token's offsets!), but the tree is still full-fidelity, and parsing is still error-tolerant.</p>



<a name="159232194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic%20parsing/near/159232194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/tree-agnostic.20parsing.html#159232194">(Feb 23 2019 at 14:53)</a>:</h4>
<p>This seems to actually work: macros can parse token-trees withoug going through toString now: <a href="https://github.com/rust-analyzer/rust-analyzer/pull/885" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/pull/885">https://github.com/rust-analyzer/rust-analyzer/pull/885</a></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>